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Eiscussion 

This  report  presents  a  discussion  of  an  approach  to  building  and 
stuiyin*  multiple  mi orocomputer  systems.  The  report  initially  discusses 
the  general  problem  area  of  multiple  computer  systems  and  indicates  why 
research  and  development  studies  need  to  be  pursued  in  this  area.  This 
report  presents  the  current  results  of  a  project  wnich  has  been  studying 
one  asnect  of  this  laree  problem.  This  particular  project  has  limited 
itself  to  the  study  of  local,  special  purpose,  multiple,  microcomputer 
architectures.  The  particular  approach  taken  has  been  to  use  a 
generalized  architecture  taxonomy  presented  by  Anderson  and  Jensen 
[&nder?6j  to  define  the  types  of  architectures  of  interest.  There  are 
ten  architectures  in  this  taxonomy.  Each  architecture  has  distinguishing 
attributes  which  will  characterize  the  functional  and  operational 
behavior  of  the  system.  The  taxonomy  only  identifies  high  level 
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attributes.  Problems  associated  with  real  time  performance,  appropriate 
methods  to  program  these  architectures,  reliability  issues,  operating 
systems  ouestions,  etc.  still  remain  to  be  solved.  It  is  felt  that  the 
existence  of  an  experimental  system  whereby  any  of  the  (functionally 
representative)  Anderson  and  Jensen  architecture  types  might  be 
conveniently  built  would  provide  a  powerful  tool  to  study  some  of  these 
problems.  This  report  presents  the  current  -esults  of  the  project  which 
has  been  concerned  with  designing  and  building  such  an  experimental 
microcomputer  system.  The  philosophy  followed  was  to  base  all  systems  on 
general  pupose  hardware  primitives  from  which  the  various  systems  can  be 
built.  It  is  critical  to  identify  these  general  concepts  so  that  the 
results  of  the  experimental  worv  can  be  extended  to  the  general  case 
where  different  hardware  and  software  implementations  occur  and  to 
permit  considering  the  newer  VLSI  devices  which  are  becoming 
commercially  available  but  which  are  rot  included  in  the  present 
experimental  system. 

Present  State  of  Knowledge 

Advances  in  VLSI  technology  have  beer,  responsible  for  the 
propagation  of  digital  technioues  for  implementing  solutions  to 
engineering  problems  which  traditionally  have  been  solved  by  other 
methods.  These  advances  have  also  made  it  possible  to  consider  building 
multicomputer  systems  where  until  quite  recently  digital  systems 
utilizing  a  single  computer  (i.e.  one  processor  in.  the  system)  were  the 
norm.  These  two  trends  are  converging  so  as  to  cause  a  growth  of 
activity  in  considering  the  application  of  systems  of  multiple  computers 
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In  a  variety  of  disciplines. 

Unfortunately,  although  it  now  seems  possible  to  build  a  multiple 
computer  system,  it  is  not  clear  how  to  build  the  best  one  in  terms  of 
matching  the  design  of  the  computer  system  to  the  needs  of  the 
application  problem.  Traditionally  the  hardware  resource  was  a  fixed 
element  and  the  software  structure  was  the  ^ajor  variable  to  consider. 
New  the  form  of  the  hardware  resource  is  also  a  major  variable.  In  fact 
it  is  not  even  clear  what  types  of  multicomputers  may  be  implemented  let 
alone  making  a  rational  choice  among  well  understood  alternatives.  There 
are  a  variety  of  single  computers  with  different  attributes.  There  are  a 
variety  of  methods  and  mechanisms  for  interconnecting  these  single 
computers  into  multiole  computer  systems.  How  does  the  designer  choose 
the  "best"  permutation  of  these  possibilities  as  a  solution  to  a 
particular  application  problem  if  the  permutations  themselves  are  not 
well  understood?  r t  is  felt  that  these  permutations  may  be  described  in 
a  systematic  manner  if  the  possible  architectures  are  characterized  by 
the  attributes  of  the  individual  computers  comprising  the  nodes  of  the 
multi  computer*  system,  the  attributes  of  the  hardware  and  software 
mechanisms  used  to  interconnect  the  individual  nodes,  and  the  attributes 
of  the  total  structure  making  up  the  global  interconnection  subsystem. 
With  this  understanding  of  the  behavior  and  attributes  of  the  possible 
multicomputer  architectures  available  as  an  information  base,  it  would 
then  be  feasible  to  progress  to  the  next  step  of  choosing  the  test 
architecture  for  a  giver  problem.  With  information  available  waich 
defines  multicomputer  architecture  types,  methods  may  be  developed  to 
guide  the  generation  of  operation  systems  and  application  programs  which 
can  then  be  matched  to  the  underlying  hardware  structure.  Any  design 
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process  involves  making  choices  among  alternatives.  Information 
describing  alternative  multicomputer  structures  is  reouirei  so  that 
those  choices  can  be  made  in  a  systematic  fashion. 

The  potential  scope  of  the  multiple  computer  architecture  space  is 
large  enough  that  a  study  of  an  important  subset  cf  the  total  space  will 
trovide  an  important  cor tributi on .  An  area  of  current  interest  is 
concerned  with  special  purpose  architectures  which  are  based  upon 
microprocessors  and  compatible  VLSI  devices.  For  the  intents  of  this 
research  project  a  “special  purpose"  multiple  computer  architecture  is 
one  whose  characteristics  are  fixed  at  design  time  while  a  'general 
purpose'  architecture  is  one  which  may  be  dynamically  reconfigured  at 
run  tine  according  to  the  needs  of  the  task(s)  it  is  supporting.  The 
subject  of  reliability  is  of  sufficient  importance  that  it  will  be 
considered  as  a  normal,  desirable  attribute  of  special  purpose  systems. 
If  a  hardware  module  failure  occurs  reconfiguration  cf  the  system  will 
be  necessary  for  the  purpose  of  maintaining  the  system  characteris ti cs 
which  were  specified  at  design  time.  Thus  at  least  this  aspect  of 
dynamic  reconfigurati on  is  of  interest  in  special  purpose  systems.  The 
special  purpose  systems  will  encompass  the  spectrun  frcn  tightly  coupled 
multiprocessors  to  loosely  coupled  systems. 

From  a  high  level  architecture  viewpoint,  a  multiple  computer  system 
may  be  described  as  a  number  of  independent  processing  nodes  which  are 
connected  together  by  an  Interconnection  Subsystem  (ISS).  Microcomputer 
based  processing  nodes  may  be  characterized  as  conventional  general 
purpose  digital  computers  which  execute  sequential  programs.  The 
attributes  of  these  nodes  may  be  described  by  existing  notations  such  as 
the  ?MS  and  IS?  descriptive  systems  [?ell7i ,Bell?8]  .  What  is  not 


5 


presently  understood  is  how  to  effectively  describe  any  arbitrary 
rrultiple  computer  architecture.  Sven  given  the  case  where  the  same 
processing  ncdes  are  used,  a  variety  of  different  multicomputer 
architectures  are  possible  whose  characteristics  are  dependent  upon  the 
structure  of  the  ISS  which  is  employed.  An  example  of  this  is  shown  in 
figure  1.  Further  permutations  are  possible  when  it  is  considered  that 
for  a  oarticular  functional  ISS  tyre  there  are  a  variety  of 
implementa  ti  cr.s  which  are  dependent  upon  the  particular  hardware 
mechanisms,  and  the  associated  standards  and  protocols  which  are 
employed.  An  example  cf  this  is  shown  in  figure  2.  Cne  must  know  the 
implementation  details  for  the  ISS  before  the  operational  behavior  of 
the  multi-computer  system  can  be  definitively  described. 

The  philosophy  of  the  ?MS  approach  to  describing  computer  systems 
is  based  upon  the  idea  that  a  fundamental  set  of  functional  building 
blocks  can  be  identified  from  which  all  digital  systems  are  comprised. 
Evidence  indicates  that  this  is  valid  for  the  uni -computer  case  and 
studies  such  as  that  which  developed  the  Cm*  architecture  [Swan77] 
indicates  that  the  approach  is  also  valid  for  the  multi-computer  case. 
Each  element  in  the  ?MS  set  is  a  functional  primitive.  Corresponding 
hardware  primitives  may  be  generated  reflecting  implementation 
considerations.  This  philosophy  can  apply  to  multiple  computer 
architectures.  A  set  of  hardware  interconnection  primitives  '.HI?),  along 
with  recognized  standards  and  protocols,  can  be  identified.  These  HI?s 
will  form  a  primltve  set  from  which  any  ISS,  within  the  context  of 
implementations  based  upon  VLSI,  are  realized. 

With  a  complete  set  of  hardware  building  blocks  known  for  the 
multi-corrpv.ter  case,  a  need  exists  for  an  underlying  structure  unor. 
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which  to  found  the  study  of  multiple  computer  architectures.  Thus  a 
rationale  exists  for  deriving  a  taxonomy  for  multiple  computer  systems. 
At  present  there  are  a  small  number  of  existing  taxonomies  [Ander76] 
[7ree?9T  [£ro72l  f’?ly?21  [Cren72l .  Of  these  taxonomies  the  one  presented 
by  Anderson  and  Jensen  appears  to  be  a  conceptually  sound  starting  point 
for  identifying  EIPs  and  ISSs.  The  nodes  ir.  the  tree  identify  high  level 
architecture  types  and  provide  an  initial  classification  of  ISSs.  From 
this  information  a  functionally  complete  set  of  ISSs  and  associated  EIPs 
may  be  defined. 

It  appears  that  a  variety  of  multiple  computer  configurations  are 
both  technically  ard  economically  feasible.  However,  detailed 
information  needs  to  be  developed  which  defines  the  total  number  of 
dis tinsuishable  architectures,  and  their  operational  characteristics 
before  design  d°cisions  can  be  made  in  a  systematic  manner. 


related  research 


Tistributed  system  taxonomies  have  been  devised  by  other 
researchers  such  as  Freeman  and  Thurber  fFree79]  .  Their  vorlt  categorizes 
local  computer  networks  into  four  classes:  packet  switched,  circuit 
switched,  bus  structure,  and  I/O  channels.  Anderson  and  Jensen  r*-nder76] 
provide  a  more  detailed  taxonomy  with  ten  architecture  classifications. 
The  architecture  taxomor.y  is  based  uoon  functional  attributes  of  the 
processing  elements  'the  nodes),  oaths  or  links,  messages,  and  switches. 
Specifically  not  considered  are  communication  strategies,  message 
addressing,  or  deadlock  situations.  The  primary  shortcoming  of  this 
taxonomy  is  that  some  actual  i mp lerer. *a t ions  of  distributed  systems  are 
hybrids  of  the  basic  taxonomy  classes.  In  addition  the  communicat ions 
orotocol  remains  as  an  important  ur.  con  side  red  attribute.  Thus  one 
attribute  of  the  ISS  which  can  greatly  affect  system  performance  is  not 
included  in  the  taxonomy.  It  m ay  be  a  reued  that  this  is  an 
implementation  consideration  rather  than  a  general  system  attribute.  A 
graphical  summary  of  the  Anderson  and  Jensen  taxonomy  is  presented  in 
figure  3. 

Several  research  projects  are  underway  on  local  computer  networks. 
Cf  particular  interest  are  the  definitions  of  various  hardware 
interconnection  primitives  ^FIPs)  and  interconnection  subsystems  (ISSs). 
although  not  referred  to  in  those  terms.  Fennels  [3enn73]  describes  a 
fault-tolerant  distributed  system  intended  for  spacecraft.  A  number  of 
modular  building  blocks  are  defined  specifically  for  the  interconnection 
of  off-the-sel'  computer  components,  '"hese  are  a  memory  interface 
■  V‘I-B3),  a  bus  interface  (31-33),  one  for  input-output  (10-33),  and  a 


so-called  "core  module"  (CORE-33).  Collectively  these  building  blocks 
constitute  a  class  of  ISSs,  at  least  for  the  particular  architecture 
investiaated  by  Fennels.  His  redundant  global  bus  approach  may  be  more 
fault-tolerant  than  is  reouired  by  most  earth  bound  networks  but 
sel f-checkir.g  components  are  an  important  area  of  interest. 

Powell  et.  al.  [Pov??]  describe  a  hybrid  architecture  with 
interesting  ISSs.  Their  network  consists  of  two  levels;  the  higher  level 
is  an  irregular  network  whose  nodes  are  interfaced  by  an  ISS  composed  of 
one  or  more  four  port  5I?s  that  collectively  make  available  an  arbitrary 
number  of  ports  to  the  network.  Only  one  of  the  ports  may  be  active  at 
ary  moment  in  time,  with  contention  control  determining  the  particular 
port.  The  lower  level  consists  of  the  node  itself  which  implements  a 
unique  fern  of  a  bus:  a  loosely  coupled  pulse  transformer.  The  advantage 
claimed  for  such  a  HI?  is  its  fault-tolerant  properties. 

4n  example  of  a  rina  topology  is  the  Distributed  Computer  System 
developed  at  UC  Irvine  [Loomis73"]  .  A  control  token  is  passed  around  the 
rir.?  for  control  Purposes.  Of  special  interest  is  their  definition  of  an 
interconnection  primitive  called  a  "local  network  interface"  or  LNI. 
This  LNI  is  intended  to  also  work  in  networks  usir.«  contention  rin 2 
control  and  in  contention  bus  networks  as  well.  Their  LNI  is  composed  of 
two  HIPs;  a  network  interface  HI?  and  a  host-specific  HIP.  The  first  HIP 
controls  data  transmission  and  reception,  recognizes  addresses,  and 
conditions  signals.  The  host-specific  HIP  is  designed  to  exchange  data 
between  the  n°twork  and  the  host  computer.  A  HI?  for  EMA  transfer  to  a 
LSI-11  UNIBUS  has  been  built. 

An  interesting  variation  on  the  IDDR  architecture  of  Andersen  and 
Jensen  is  the  MICRONET  system  [Wit??-!  .  Ir.  this  system  each  node  is 
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connected  to  a  horizontal  and  to  a  vertical  bus  through  a  RIP  defined 
for  this  system.  Each  node  consists  of  a  local  computer  (a  C)  and  a  HI? 
which  imolements  the  bus  connections  and  which  interacts  with  the  local 
C.  In  a  given  configuration  the  local  C  can  be  eliminated  and  a  given 
node  may  only  consist  of  the  special  HI?  which  maintains  the  ISS  at  a 
given  bus  ’unction.  *n  interesting  facet  of  MICRONET  is  that  it 
resembles  the  IDER  architecture  but  it  has  an  important  difference  in 
that  any  PICRONET  node  can  communicate  with  any  other  node  connected  to 
a  bus  while  in  the  IEER  case  a  given  node  can  only  communicate  with  a 
neighbor  ir.  the  array.  This  difference  affects  the  switching  complexity 
of  the  system,  its  reliability,  and  other  system  attributes.  Such 
differences  must  be  resolved  in  a  generally  complete  taxonomy. 

As  mentioned  earlier  a  communications  protocol  is  an  important 
aspect  of  a  distributed  system  because  of  its  infuence  upon  the  system's 
performance.  At  the  "common  carrier"  network  level  much  standardization 
is  underway  by  various  international  and  national  institutions.  A  local 
computer  network  benefit  by  using  standard  components.  Standards  are 
being  discussed  at  seven  different  levels,  "-any  or  most  of  which  may  be 
important  to  the  context  considered  here.  It  is  hoped  that  someday  both 
hardware  and  software  modules  may  be  commercially  available  at 
relatively  low  cost  (compared  to  the  cost  of  producing  "optimal"  modules 
for  a  Darticular  application  problem'.  A  recent  technical  journal  is 
devoted  to  network  protocols  (September  19?9  Computer,  Special  Issue  on 


Network  Protocols ) . 
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Current  Results  from  Project  :  Modular  Microcomputers 

An  example  cf  a  modular  microcomputer  system  is  the  Microprocessor 
Independent  Microcomputer  (MIMIC)  fcarpy77a,  Var79l .  This  is  based  upon 
a  standardized  nicroprocesso r  bus,  and  a  set  of  modular  hardware 
primitives  which  are  compatible  with  any  of  the  current  generation  of  6 
bit  microprocessors .  Th®se  systems  are  being  extended  to  the 
^ul ticomouter  case  by  defining  hardware  primitives  from  which  a  variety 
cf  systems  may  be  realized.  A  first  set  of  hardware  interconnection 
primitives  'HIP)  have  been  designed  and  implemented  which  include  (1)  a 
shared  memory,  (2)  multichannel  parallel  and  serial  links,  (3)  a 
multiport  bus  window,  (5)  an  intelligent  channel  processor,  and  (5) 
extensions  of  the  MIMIC  bus  standard  to  support  shared  bus  systems. 
These  5I?s  are  beirg  installed  in  a  three  node  MIMIC  system  so  as  to 
perform  experiments  on  a  variety  of  interconnection  subsystems  ( IS S )  . 
The  selection  of  the  particular  hardware  interconnection  primitives  was 
done  by  an  analysis  of  a  high  level  taxonomy  [».nder?Sl  .  The  designs  were 
a  function  of  the  characteristics  of  the  MIMIC  bus  and  currently 
available  VLSI  devices.  It  is  felt  that  the  experience  gained  with  such 
a  system  will  oroviie  valuable  insights  into  understanding  more  complex 
systems  based  uoon  future  VLSI  devices  and  more  complex  processing  nodes 
and  interconnections  subsystems. 

Where  the  MIMIC  multicomputer  system  is  based  upon  internal 
University  of  Connecticut  designs  and  8  bit  microprocessors,  a  second 
multiple  computer  based  upon  commercial  modules,  and  the  16  bit  LSI  11 
microorocessor ,  has  also  teen  designed  and  implemented.  The 
interconnection  subsystem  (I3S)  is  entirely  based  upon  serial  links. 
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Although  this  is  a  very  low  pe~fornar.ce  environment  the  particular  ISS 
is  very  flexible,  permits  a  variety  of  architectures  to  be  implemented 
urder  program  control,  and  is  currently  being  used  to  explore  problems 
with  distributed  software  systems. 

Current  Pesults  :  Hardware  Interconnection  Technioues  and  Primitives 

There  are  a  limited  number  of  ways  that  digital  hardware  modules  and 
systems  may  be  interconnected.  This  limitation  is  based  upon  the  nature 
of  digital  hardware  itself.  Additional  limitations  occur  if  only  certain 
types  of  digital  systems  are  considered  such  as  those  which  are  based 
upon  microprocessors  and  associated  VLSI  devices.  In  addition  to 
hardware  limits  functional  aspects  must  be  considered.  In  the  ?MS  system 
[Hell'll]  there  are  two  interconnec tion  primitives?  the  linx  (L)  and  the 
switch  'S'.  This  is  a  functional  definition  and  in  practice  other 
primitives  such  as  the  memory  (M),  the  controller  X),  etc.  may  serve  to 
i^ple'-ent  an  interconnection  function.  To  say  that  something  is  an 
interconnection  mechanism  reouires  first  stating  the  level  of  viewpoint 
which  is  being  used.  T^r  this  paper  at  least  thre°  viewpoint  levels  are 
employed.  These  levels  are  the  '1)  electronic  hardware,  (2)  system 
functional,  and  (3)  software  functional. 

At  the  electronic  hardware  level  in  microcomputer  based  systems 
only  four  primary  mechanisms  exist  for  interconnecting  modules.  These 
are  (1)  bit  serial  transmission  via  a  serial  lin*  under  the  control  of  a 
processor,  (2)wcrd  parallel  transmission  via  a  parallel  linix  under  the 
control  of  a  processor,  (3)  word  carallel  transmission  over  a  bus  (i.e. 
the  internal  microcomputer  bus),  and  '4)  word  parallel  transmission 
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through  storage  in  a  shared  memory  (I/O  devices  may  he  viewed  as  a 
shared  memory).  These  are  the  basic  possibilities  with  hardware 
mechansims  of  higher  functionality  being  constructed  from  this  set  of 
techniques . 

At  the  system  functional  level  the  P*IS  concepts  appear  to  be 
complete.  Therefore  a  given  approach  will  be  characterized  as  a  ?MS 
primitive.  An  interconnection  mechanism  may  have  a  completely 
programmable  computer  within  it  but  if  the  function  it  is  performing  is 
that  of  a  switch  then  a  switch  it  is. 

At  the  software  functional  level  the  activities  that  the  hardware 
system  is  supporting  must  be  considered,  ^clayien  [tfcfay?6)  suggests 
such  a  functional  viewpoint  by  proposing  the  "functional  layering" 
concept  for  organizing  software  systems  in  multicomputer  systems.  In 
this  approach  the  functional  responsibilities  of  the  network  nodes  are 
divided  into  three  major  layers  which  are  (1)  transmission  management, 
(2)  node  functional  task  management,  and  (3)  application  operations. 
This  identifies  at  least  three  node  tasks  which  will  have  a  time  ordered 
relationship  with  resulting  implications  about  the  potential  parallelism 
( concurrency)  of  software  operations  associated  with  the  node.  The 
performance  cf  the  node  can  be  immediately  affected  by  the  hardware 
resources  in  the  node  which  are  available  to  support  (exploit)  this 
potential  parallelism. 

A  further  classification  of  the  interconnection  mechanisms  can  be 
made  if  the  system  is  divided  into  the  two  broad  classes  of  intemodal 
and  intranodal  connections  with  intemodal  being  those  connections 
between  disjoint  nodes  and  intranodal  being  those  connections  within  a 
distinct  node  in  the  system.  At  first  it  might  be  tempting  to  envision 


that  fundamentally  different  hardware  mechanisms  are  necessary  to 
realize  interr.odal  versus  intranodal  connections.  However  this  is  not 
true.  As  stated  th^re  are  a  severely  limited  number  of  basic  digital 
hardware  possibilities  for  imolementing  hardware  connections  and  these 
cover  both  the  inter  and  intra  nodal  cases.  The  same  hardware  modules 
may  be  employed  to  realize  both.  The  dlstlction  becomes  a  matter  of 
functional  deflation  as  is  true  with  the  P^S  primitives. 

With  the  above  discussion  as  ar  explanation  let  the  following 
hardware  interconnection  primitives  be  defined  for  microcomputer  based 
sys  terns . 

Ceneral  microcomputer  hardware  interconnection  primitives: 

1 .  Microcomputer  bus  (MB)  :  primary  ir t ranoia 1  correction  between 
the  microprocessor  and  all  other  functional  modules  in  the  node. 

2 .  Microcomputer  bus  window  (M3W)  ?  electronic  "-appir.*  between  MBs . 

3. Shared  Memory  (Sm)  :  a  memory  which  "■av  be  accessed  by  mere  than 

one  microcomputer  or  microprocessor. 

4. Shared  Microcomputer  Bus  (SM35  t  an  internal  bus  which  may  be 
dynamically  time  shared  by  two  or  T*re  mi  coprocessors . 

5.  Shared  Communication  Bus  :=  fSCB)  : =  an  external  bus  which  is  share! 
by  two  or  more  microcomputer?  for  internoial  communication 

6. Serial  Link  (Ls)  :  a  bit  serial  interface  whose  operations  are 
controlled  by  a  microprocessor.  Normally  only  used  for  transmission 
of  information. 

7. Parallel  Link  ( L p ^  :  a  word  parallel  interface  whose  operations  are 
controlled  by  a  microprocessor.  May  be  usei  for  simple  message 
transmission  or  as  part  of  a  control  mechanism. 

?  .Tra r.smiss i on  Processor  < ? t )  :  a  microprocessor  whose  puroose  is  to 


support  the  transmission  layer  within  a  node  or  between  nodes.  A 
functional  division  into  two  senarate  types  cf  ?t  may  he  necessary 
if  internodal  and  intranodal  transmission  operations  prove  to  he 
fundamentally  different. 

2.  DMA  controller  'Kdma)  :  implement  dma  intranodal  transfers. 

There  are  other  possibilities  such  as  an  alignment  network  but  upon 
examination  these  seem  to  be  constructed  from  the  above  primitives. 

Siven  these  basic  hardware  techniques  the  next  level  is  to  identify 
hardware  interconnection  primitives  which  may  be  used  to  realize  any 
hardware  architecture  of  interest.  In  this  study  the  Anderson  and  Jensen 
taxonomy  was  employed  to  define  such  an  architectire  space.  As  mentioned 
in  the  introductory  discussion  other  architectures  seem  to  be  possible 
but  this  taxonomy  seems  to  be  representative .  Therefore  the  basic  set  cf 
hardware  techniques  were  used  to  define  HIPs  from  which  any  of  the 
Anderson  and  Jensen  architectures  could  be  realized.  What  is  of  interest 
is  that  only  a  small  set  of  HIPs  is  reouirei.  These  are  as  follows. 

MIMIC  hardware  interconnection  primitives  : 

l.Four  channel  serial  link  fS;Ls»4)  :  dual  mods  to  support  internodal 
cr 

intranodal  information  transfers. 

2. lour  channel  parallel  link  (SiLpi4'  :  dual  mod?  to  support  internodal 
or  intranodal  information  transfers. 

3.  Four  channel  bus  window  :  address  mapping  is  programmable  to 

support  a  variety  of  bus  window  based  architectures. 

4.  Transmission  computer  (CtJdma)  :  a  senarate  computer  with  a  Ip 

and  a  Kdma  to  the  MI^IC  bus  and  which  may  control  either  the 
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S;Ls;4  or  the  S;Lp;4.  This  will  support  either  internodal  or 
intranodal  software  "transmission  layers."  Ir.  this  KIP  tae  CtJima 
will  support  either  t ransmiss ion  through  a  S;lsi4,  a  S;lpJ4,  or 
it  provides  a  dma  block  transfer  capability  within  a  node.  Cost 
modularity  considerations  indicated  the  efficiency  of  grouping  these 
functions . 

Jach  of  these  modules  has  been  designed  and  constructed  for  use  in 
the  MIMIC  systems.  Since  the  MIMIC  system  is  representative  of  systems 
such  as  the  S-130  based  microcomputers  [Elm?3]  the  results  of  this 
implementation  are  applicable  to  3-l?P  type  systems  also. 

Current  Results  :  MIMIC  Multicomputer  Architectures 

The  following  figures  oresent  the  particular  implementations  of  the 
MIMIC  HIPs  defined  above,  some  of  the  interconnection  subsystems 
possible  with  these  HIPs,  and  some  of  the  resultant  multi  microcomputer 
architectures.  Figure  4  presents  a  bloc>  diagram  of  the  basic  four 
channel  link  HIP.  This  diagram  represents  both  the  3;Lp?4  and  the 
S *L s * 4 .  Figure  4(b)  and  4(c)  show  a  ?"S  diagram  version  of  these  HIPs. 
The  PMS  representation  is  much  clearer  and  compact  as  a  g-aphical 
-actant  ion .  An  interesting  point  is  that  although  the  functional 
representations  of  the  the  SJL?54  and  the  3JLS54  modules  are  the  same 
their  effects  on  the  operational  characteristics  of  the  system  are 
significantly  different.  Ps  an  exampl®  physical  distance  and 
transmission  speeds  are  ouite  different  and  the  S:Lp?4  may  also  serve  in 
a  variety  of  control  operations  which  are  impossible  to  the  S«LS'4.  This 
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is  a  vivid  illustration  of  the  need  to  examine  the  implementation 
considerations  beyond  the  functional  definitions  to  see  the  important 
effects  of  using  different  interconnection  mechanisms. 

Figure  5  presents  a  PMS  diagram  of  a  typical  microcomputer  such  as 
a  MIMIC  or  S-130  based  machine.  Figure  6  presents  a  PMS  diagram  of  the 
transmission  processor  HI?  with  the  various  module  features  defined. 
Note  that  this  is  auite  similar  in  nature  to  the  machine  of  Figure  5  tut 
the  buffered  bus  connections  ard  the  Lp  and  the  Sima  provide  the 
hardware  means  to  realize  the  transmission  functions. 

Figure  7  combines  the  modules  of  figures  4,5  ard  6  to  show  a  typical 
node  which  consists  of  a  node  microcomputer  and  the  HIPs  necessary  to 
realize  a  variety  of  interconnection  operations.  This  is  a  general 
purpose  structure  which  will  permit  implementing  a  number  of  the 
architectures  from  the  Anderson  an*  Jer.sen  taxonomy. 

Figure  6  presents  a  block  diagram  of  the  four  channel  bus  window  and 
a  corresponding  PMS  represntation.  This  HIF  actually  realizes  four 
independent  bus  window  channels  in  a  single  hardware  module.  This  was 
done  for  reasons  of  cost  modularity  as  it  was  reasonable  to  multiplex 
the  3W  control  and  timing  functions  among  separate  bus  window  channels. 

Figure  9  presents  two  possible  architectures  feasible  with  the  3W54 
HIP.  In  figure  9  a)  is  shown  a  three  node  configuration  which  permits 
either  a  DDL  type  machine  or  a  true  IDS  configuration.  The  address  maps 
in  the  bus  window  are  programmable  which  permits  defining  a  mapping  of 
bus  window  address  spaces  which  functionally  define  different 
architectures.  As  an  example  each  node  might  have  to  manage  the  ring 
messages  through  a  store  and  forward  mechanism  if  the  3W  address  maps 
only  oermitted  unidirectional  information  transfer.  In  another  case  the 
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5W  adress  raps  may  be  set  to  permit  any  C  In  the  network  to  access  any 
other  C  indirectly  through  the  B'*s.  Ir.  these  cases  the  functional 
software  operations  and  system  behavior  will  differ. 

Figure  1?  presents  a  four  node  nine  implementation  of  the  structure 
originally  shown  in  figure  2(b).  In  this  case  the  configuration  of 
figure  7.  which  employed  the  Ct  and  S;LsJ4  HIPs,  is  used  to  build 
higher  performance  DDL  architecture.  Other  basic  structures  are  also 
possible  from  this  fundamental  node. 

Present  studies  show  that  nine  of  the  ten  possible  Anderson  and 
Jensen  architectures  are  possible  from  these  hardware  primitives.  Only 
the  ICSB  structure  is  not  possible  using  these  mechanisms. 

These  systems  reflect  the  possible  multicomputer  hardware  structures 
possible  from  a  fundamental  set  of  building  blocks.  The  next  phase  of 
the  study  will  define  system  level  characteristics  which  are  of 
interest.  These  system  attributes  will  then  be  examined  in  light  of  a 
particular  EIP/'ISS  implementation  to  draw  conclusions  about  the  effects 
of  particular  hardware  techniques  upon  system  level  behavior.  From  this 
information  additional  studies  will  exolore  developing  "ethods  for 
matching  the  best  hardware  architecture  to  the  reeds  of  a  given  design 
problem . 

Current  Results  :  Commercial  i^ulti^icrocomputer  systems 

Where  the  MIVIC  hardware  orimitives  reflect  specialized  design  whose 
primary  prupcse  is  a  consideration  of  hardware  architectures,  another 
facet  of  the  project  being  presented  here  is  concerned  with  distributed 
software  issues.  It  is  not  presently  clear  what  are  the  relationships 
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between  the  structure  of  a  distributed  software  system  and  the 
underlying  distributed  hardware  structure.  To  study  this  and  to  permit 
focusing  upon  the  distributed  software  issues  an  eight  node  LSI11 
distributed  microcomputer  has  been  designed  and  is  being  constructed. 
This  system  has  a  general  purpose  ISS  whose  structure  may  be  varied 
under  program  control.  The  ISS  is  constructed  from  serial  link  HIPs  to 
provide  full  point  to  point  communication  between  any  node  in  the  system 
or  any  permutation  of  interest.  The  software  system  defines  the 
particular  ISS  structure  which  is  to  be  used.  This  permits  an  easy  and 
flexible  shifting  between  different  fundamental  architectures  as  the 
interests  of  the  software  research  efforts  specify  a  change. 

Some  examples  of  this  system  are  shown  in  figures  12  throueh  12.  The 
basic  system  is  shown  in  figure  10  (b)  which  consists  of  eight  L5I11 
nodes,  as  defined  by  fieure  12  (a),  a  full  point  to  point  serial  link 
ISS,  and  a  PI!?  11/60  for  program  development  support.  An  example  of  a 
full  point  to  point  133  is  shown  in  fisure  11  (c).  Any  node  can  directly 
communicate  with  any  other  node  through  dedicated  serial  links.  This  is 
a  completely  *eneral  scheme  with  some  of  the  possible  permutations  shown 
in  figures  11  a),  ll(b1,  ll(d^,  Lite',  and  12.  The  software  system 
defines  the  dedicated  serial  links  which  it  chooses  to  use  and  that 
defines  the  ISS.  As  can  be  seen  this  permits  DDL,  ICDS ,  ICDL,  DDC ,  IDDX , 
and  IDDI  architectures  which  are  six  of  the  ten  basic  Anderson  ar.d 
Jensen  structures.  This  will  provide  a  great  flexibility  to  support 

experimental  work  with  distributed  software. 

) 

To  show  other  features  of  this  system  figure  12  presents  a  dual  DDL 
system,  basically  this  is  a  system  with  ar  alternate  path  to  compensate 
for  a  failure  of  a  given  dedicated  link  or  the  node  which  supports  that 


Unit.  Such  a  system  provides  an  experimental  vehicle  for  considering 
reliability  and  software  recovery  issues.  In  this  system  two  DDL  loops 
are  physically  defined.  During  normal  operations  only  ar.  active  DDL  path 
will  he  employed.  In  case  of  a  link  failure  an  alternative  link  will  he 
activated.  Ir.  case  of  a  node  failure  the  r.ode  tasks  need  to  he 
tranferrei  to  ar  active  rode  for  a  degraded  mode  of  operation  and 
associated  alternative  links  will  become  active. 

Additional  system  architectures  are  possible  from  this  general 


purpose  serial  lir.k  ISS. 
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Conclusions 

It  seems  to  be  possible  to  identify  a  set  of  architecture  types  which 
may  be  used  to  characterize  and  study  multiple  computer  systems.  It  also 
seems  possible  to  identify  the  fundamental  hardware  techniques  which  are 
employed  in  implementing  hardware  interconnection  mechanisms.  These  two 
results  have  been  combined  in  this  project  to  identify  an  initial  set  of 
hardware  interconnection  primitives  (KI?s)  which  may  be  used  to  build 
functional  representative  interconnection  subsystems  (ISSs)  which  result 
in  a  distinguishable  architecture  type.  An  initial  set  of  these  HI?s 
have  been  designed  and  constructed  to  permit  the  convenient  construction 
of  the  various  multiple  microcomputer  architectures  of  interest. 

There  are  a  number  of  major  issues  which  remain  to  be  studied  prior 
to  the  development  of  well  ordered  design  procedures  concerning  multiple 
microcomputer  systems.  Car.  the  types  of  all  possible  architectures  be 
identified  and  characterized  in  a  meaningful  way?  The  :.r.derson  and 
Jensen  taxonomy  indicates  that  this  is  possible  but  a  more  complete 
taxonomy  is  still  necessa-y  which  will  also  consider  implementation 
attributes.  Can  an  encompassing  set  of  functional,  modular  primitives  be 
identified  which  support  the  implemer tati on  of  any  multiple  computer 
syste-"?  This  project  has  accomplished  this  seal  within  a  limited 
context-.  This  successful  result  indicates  that  suc>'  a-  objective  "ay  be 
generally  possible.  The  ?*S  system  proviies  a  conceptual  base  for 
identifying  primitives  In  unicomputers.  It  remains  to  be  seen  whether 
this  can  be  done  for  multiple  computers.  Toes  an  appropriate  description 
language  exist  which  can  be  used  to  describe,  simulate,  ar.d  manipulate 
the  architectures  of  multiple  computer  systems  as  a  support  for  the 


desia.o  process?  Ir.  a  uni  com  outer  the  architecture  is  often  cnaracter  lzei 
by  either  the  structure  of  the  processor  (stack,  register  oriented, 
etc.)  or  by  the  processor  instruction  set  itself  [Amd64,  3rovn7S]  .  What 
is  the  proper  way  to  characterize  the  multiple  computer  architecture? 
Perhaps  the  best  way  is  to  characterize  such  systems  in  terms  of  their 
interconnection  sihsystems  which  must  eventually  encompass  both  the 
hardware  subsystem  and  the  software  subsystem.  What  are  the  important 
attributes  of  this  overall  int'=rccn-ecti on  subsystem?  What  are  the 
tradeoffs  between  hardware  and  software  mechanisms?  What  is  the 
relationship  between  the  distributed  hardware  system  and  the  distributed 
software  system?  These  and  other  such  Questions  regain  to  studied  and  to 
be  answered.  It  is  felt  that  the  identification  of  the  hardware 
primitives  and  the  existence  of  the  experimental  system  described  in 
this  report  will  proviie  an  effective  experimental  means  to  examine  some 


Figure  1  (a)  An  example  of  a  high  speed 
interconnection  subsystem  using 
bus  windows  and  a  shared  memory 

System  characteristics 

1.  Tightly  couple 

2.  High  performance  133 

3.  Nodes  are  physically  close 

4.  Expensive  IS3 

5-  Higher  reliability 

6.  Oood  modularity 

7.  Central  OS  kernel 

3 .  etc  ? 


Figure  Kb)  An  example  of  a  low  speed 
interconnection  subsystem  using 
serial  links. 

System  characteristics 

1.  Loosely  coupled 

2.  Low  performance  ISS 

3.  Nodes  may  be  physically  far 

4.  Inexpensive  ISS 

5.  Low  reliability 

6.  Good  modularity 

7-  Distributed  OS  kernel 

S.  etc.  ? 


Figure  2  (a)  DDL  architecture(Ander76) 
in  which  node  ?  must  support  both  ring 
message  transmission  and  local  tasks 


Figure  2  (b)  DDL  architecture  { Ander?^' 
with  switch  S  supporting  ring 
message  transmission  and  node  ? 
supporting  local  tasks. 


a)  DDL  :  Direct 
Dedicated  Loop 


b)  DDC  :  Direct  c)  DSM  :  Direct 

Dedicated  Complete  Shared  Memory 


d)  DSB  :  Direct 
Shared  Bus 


Remote 
Bus  Window 
Interface  :  #1 


Remote 
Bus  Window 
Interface  :  #2 


Remote 
Bus  Window 
Interface  :  #3 


Remote 
Bus  Window 
Interface  :  #4 


4  channel  bus  window  switch 


Bus  Window  Controller 

1.  Address  Mapping 

2.  Synchronization 

3.  Timing 


Local 

Bus  Window 


Interface 


LBW  :  to  MIMIC  3us 


Figure  3a  31ocic  diagram  of  four  channel  Bus  Window  HI? 
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where 


Figure  3b  ?M3  diagram  of 
4  channel  bus  window  HI? 


Figure  Sc  Reduced  ?MS  diagram 
of  4  channel  bus  window  HI? 
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Figure  9(a)  Three  node  system  with 
Bus  Window  HIP  Implementing  ISS. 
Possible  architectures  are  DDL  If 
window  addresses  are  set  so  must  do 
"store  and  forward"  to  pass  messages, 
and  IDS  If  window  addresses  are  set 
so  ary  C  may  access  part  of  address 
space  of  any  other  C 


C  C  C 


RBW  RBW  RBW 


R3W  RBW  P-BW 


I/O.  M  . 

shared  shared 

Figure  9  (b)  Three  node  system  with 
Bus  window  HI?s  acting  to  establish  a 
DSM  architecture.  Window  addresses  are 
set  so  each  C  can  access  any  part  of  the 
shared  M. 

C 


Figure  1C  Four  node  DDL  architecture  with  the  two  rUPs  (1)  Transmission 
Processor,  and  (2)  Four  charxiel  serial  link  being  used  to  implement 
the  ISS.  This  is  a  specific  implementation  of  the  general  structure 
shown  in  figure  2  (b).  Note  that  each  C.  1  HIP  combination  results 
in  two  unused  serial  links  pemode.  s  These  may  be  used  to 

support  other  system  features  or  provide  backup  links  for  reliability 
purposes . 


Figure  (a)  Typical  LSI11  Node; 
serial  links  (Lj  used  for  full 
point  to  point  a  interconnection 
in  3  node  system 


Figure  10(c)  Distributed  13111 
System  with  full  point  to  point 
interconnection  as  oer  figure 
1C  (c) 


Figure  11  (d)  IDDF.  from  13111  3ystem 


Figure li  (e)  IDDI  from  13111  system 
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